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The application of a well-known and fairly simple 
information retrieval technology to the process of testing 
undergraduate college students is described. The Computerized Test 
Library allows the student and/or the professor to extract questions 
of varying difficulty covering whatever topic the student is 
studying; if the student performs at. the appropriate level, both the 
student and professor are so informed. The Computerized Test Library 
is created from punched card record groups (a single questions and 
its coded information) on a 2311 disk pack, comprising two 
99-cylinder sequential files. The disk records have a length of 80 
bytes and are blocked at 3600 bytes. The punched card record groups 
consist of a header card and a variable number of questions and 
answer cards. The package is comprised of a total of six programs 
written in PL/1 language. (The programs and further documentation are 
available from Jon Gosser, Psychology, Delta College, University 
Center, Michigan U8710.) Card Input Formats, Description of Codes in 
Computerized Test Library, examples of test questions, answers, and 
related data, instructions as to file; maintenance, and a brief 
evaluation of the tests of the Computerized Test Library c:re 
provided. (DB) ' 
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This paper will describe the application of well-known and fairly simple informa- 
ijL/ tion retrieval technology to the very common and complicated process of testing students 
in undergraduate college courses. The retrieval of information about test, questions and 
the retrieval, addition, deletion, alteration, and the rearrangement of test questions 
(essay or objective) is a common problem to all professors. The system described below 
does not make the job of the professor any easier, but it does allow him to be more 
confident that his tests are measuring his student's learning in a reliable and valid 
manner. 

In addition to allowing the improvement of test reliability and validity the 
^ computerized test library deals directily with t^vro other major problems in college 

teaching (1) accountability or measurable objectives and (2) allowing the student to 
proceed at his own pace. It deals with accountability by allowing the professor to 
C^y^ store and retrieve information on the performance of his students on individual questions 
and on objectives that are tested by individual questions.^ 

The test library allows either the student and/or the professor to extract questions 
psdlTi of varying difficulty covering whatever topic the student is studying and if the student 

performs at the appropriate level to inform the student and the professor of this. If 
the student does not per:^orra well enough, he can restudy and then take an equivalent^ 
test on the same topic. Thus, the student can proceed with the course objectives at his 
own rate* The test library does not, at present, deal with the many and significant . 
objectives of college courses that cannot as yet be reduced to test questions. 

^Paper presented on April 7, 1973 at Michigan Academy of Science, Arts and 
Ann Arbor, Michigan 
Assistant Professor, Psychology 
^^Director of Data Processing 
•^^Programmer Analyst 

■'Delta College has fully supported this as well as many other uses of its 

computers in teaching, 
^determine which questions test a given objective can be done by face validity 
Q (professor opinion) or by more refined statistical techniques. 

^opinion of professor or by appropriate statistical techniques. 



Creation of Library 

The computerized test library is created from punched card record groups (a 
single question and its coded information) on a 2311 disk pack- comprising of two 
(2) 99-cylinder sequential files. The disk records have a length of 80 bytes and 
are blocked at 3600 bytes. 

New question number record groups may be added to the files (with a question 
number higher than the last one used) or the file can be r^'-created from the original 
punched card record groups. 

The source files records may be punched back into card format, with sequence 
numbers on the question/ansx^er cards. 

The punched card record gr.iups consist of a header card, and a variable amount 
of questions and answer cards. 



The package comprises a total of six (6) programs written in the PL/I 
language. These programs and further documentation can be obtained from 
Jon Gosser, Psychology, Delta College, University Center, Michigan, 48710. 
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Card Input Formats 
Header card 
cc 

1-5 question number 

6- 10 answer codes (maximum of five (5) answers) 
11-78 see description of codes section 

79-80 blank 
Quiistion/answer cards 
cc 

1-5 question numbers (same number as header card) 
6 blankv 

7- 77 question/answer content 

78-80 card sequence number (increments of 10) 



! 
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DESCRIPTION OF CODES IN COMPUTERIZED TEST LIBRARY ^ 
September 1, 1972 



Code 1 Spaces Allowed - 5 This is the reference number of each ques- 

tion and is assigned by Data Processing. 
This number can never be changed and when 
you want a question printed out, this is 
the number you have to specify. 

Code 2 Spaces Allowed - 3 This is the major content area which the 

question is designed to teach. 

1. I = Introduction, History and 

Systems 

2. VL = Human Verbal Learning 



3 . L = Learning 

4. M = Motivation and Emotion 

5. SP = Sensation and Perception 

6. p = Physiological 

7. ST = Statistics, Measurements and 

Testing 

8. PT = Personality and Clinical 



9. C = Child 



10. S = Social 



Code 3 Spaces Allowed - 4 



Code 4 Spaces Allowed - 3 



11. TP = Thinking and Problem Solving 

The refined content area the question is 
designed to test; for example, code 2 
might contain L representing the major 
content area of learning while code 3 
might refer to discrimination learning 
only. Specific codes not developed as 
yet. 

Type of question 

1. K = Knowledge and Facts 

2. IAS= Intellectual Abilities & Skills 

3. AP = Application 
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4. AN = Analysis 

T 5 . S = Synthesis 

^These codes are examples of Psychology. 

In the programs these are character fields and the professor can use any 

system of codes he wishes without changing any of the programs. 



Code 5 



Spaces Allowed - 3 



How important the question is to Psychology. 
1 = the most important; 2 = next; 3 = next; 
4 = next; 5 = least important. 



Code 6 Spaces Allowed - 4 Discrimination Index 

The discrimination index indicates how the 
high scoring students (top 25%) on the whole 
test performed on a specific question as com- 
pared to low scoring students (bottom 25%) 
performed on a specific question. The index 
varies between -1 to +1. A -1 indicates that 
? all low scoring students answered the question 

correctly and none of the high scoring students 
answered it correctly. A 0 indicates than an 
equal number of high scoring and low scoring 
students answered the question correctly. A 
+1 indicates that all the high scoring stu- 
dents answered the question correctly and 
none of. the low scoring students answered it 
correctly. 



Code 7 



Spaces Allowed - 3 



Difficulty Index 



The difficulty index indicates the per cent 
of the students who answered the question 
correctly. Only the top and bottom 25% of 
the students on the test are included. 
These are the same students who were included 
in calculation of the discrimination index 
above. 



Code- 8 



Code 9 



Code 10 



Code 11 



Code 12 



Spaces Allowed 



Spaces Allowed - 3 



Spaces Allowed - 5 
Spaces Allowed - 3 

Spaces Allowed - 3 



This is the date of the last semester in 
which the question was used and is replaced 
every semester that the question is used. 

This identifies questions which are used 
for pre-tests, sample tests and other 
questior.s which do not directly count on 
the student's grade. 

This is the total number of times the ques- 
tions have been used. 

Professional Approval Code 

1 = Department Approval 

2 ■= "Delete'* or "No" Opinion 

This is the course used when identifyitfg the 
course for which the questions are designed 
to be used . 

GP = General Psychology 
AP = Applied Psychology 

C = Child 
PER = Personality. 
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Code 13 



Code 14 



Code 15 



Code 16-22 



Spaces Allowed - 4 



Spaces Allowed - 4 



Spaces Allowed - 3 



Spaces Allowed - 3 
for each code 



This is the date on which the- questions were 
punched and is assigned by Data Processing. 
This date never changes. 

This is a code identifying the source from 
which the questions are taken. (supplied 
upon request:) . 

Number of corrections, discrimination index 
by .1. or more and/or difficulty index by. 10 
or more. 

Not yet assigned. 
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Question Extraction 

Requested test questions are selected by card input. An internal program sort 
places the question numbers in ascending sequence so the selected question number record 
groups can be written on an indexed sequential disk file. The record groups are then, 
printed in the original card input sequence.. The instructor's copy (see Figure 1) con- 
tains all the various fields in the header record plus the question, and answer while 
the student copy (see Figure 2) consists only of the question, and answer choices. The 
student copy can be directly reproduced for use on a test. An option of printing a new 
(2) line descriptive title at the top of each page is available. 

Card Input Format 

cc 

1-5 question number 
6-80 blank 

The files can be sorted by the header record fields for a total of thirty (30) 
bytes. An extract program reads the source files, extracts the desired fields, and 

i 

writes a i04-byte record on another disk pack. The 104-byte records are then sorted and 
the print-out may commence with a chosen question number record group. 

This "capacity to sort by any of the codes or combinations of five or more codes allows 
the . prof essor to select questions covering a given major or minor content area and have 
the question printed in order of difficulty, (code 7) importance, (code 5) degree of 
discrimination (code 5), and number of time used. This allows the selection of questions 
that meet predetermined criteria set by the professor or the student. Obviously one can 
select questions by any of the other codes or combination of one or more of these codes. 
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FIGURE I 



MARCH 29, 1973 DELTA COLLEGE PAGE 1 

1. AMBIVALENCE ILLUSTRATES 10 

1. FRUSTRATION TOLERANCE. 20 

♦* 2. APPROACH-AVOIDANCE CONFLICT. 30 

3. APPROACH-APPROACH CONFLICT. ^0 

4. AVOIDANCE-AVOIDANCE CONFLICT. 50 

5. 60 
(1) 352 i2) M (3) ih) K (5) 3 (6) (7> (8) 372 
(9) (10) 2 (II) 1 (12) GP (13) 968 (14) 5P (15) 

116) (17) (18) (19) (20) (21) (22) 

2. THE DESIRE FOR AN OBJECT FOR ITS OWN SAKE IS KNOWN AS.. 10 

1. EXTRINSIC MOTIVATION 20 

2. POSITIVE TRANSFER 30 

3. STRESS INCENTIVE 40 
♦* 4. INTRINSIC MOTIVATION. 50 

(1) 1201 (2) M (3) (41 K (5) 2 (6) (71 (8) 372 

(9) (10) 2 (11) 1 (12) GP (13) 969 (14) 14P (15) 

(16) (17) (18) (19) (20) (21) (22) 

3. THE AMOUNT OF STRESS WHICH A PERSON CAN BEAR WITHOUT SHOWING BEHAVIORAL 10 
DISORGANIZATION IS HIS 20 

1. LEVEL OF ASPIRATION 30 

** 2. FRUSTRATION TOLERANCE , 40 

3. AVOIDANCE LEVEL ' 50 

4. PHYSIOLOGICAL LIMIT 60 

5. POINT OF INTERESCTING GRADIENTS 70 
(1> 923 (2) M (3) (4) K (5) 2 (6) (7) (8) 372 
(9) (10) 2 (11) 1 (12) GP (13) 869 (14) 8P (15) 

(16) (1?) (18) (19) (20) (21) (22) 

4. THE SYMPATHETIC AND PARASYMPATHETIC SYSTEMS ARE DIVISIONS OF THE 10 

1. PRI MARTY NERVOUS SYSTEM. 20 

*« 2. AUTONOMIC NERVOUS SYSTEM. 30 

3. EPINEPHRINE-NOREPINEPHRINE SYSTEM. 40 

4. GENERAL NERVOUS SYSTEM., 50 
(1) 138 (2) P {3) (4) K (51 3 (6) .40 (7) 80 (8) 971 
(9) (10) 1 (11) 1 (12) GP (13) 768 (14) 2P (15) 

(16) (17) (18) (19) (20) (21) (22) 

5. THE TWO HEMISPHERES OF THE CEREBRUM ARE CONNECTED BY 10 

1. THE BRAIN STEM, 20 

2. THE RETICULAR FORMATION. 30 
** 3. THE CORPUS CALL0S13M. 40 

4. A LARGE FISSURE. 50 

(1) 45 (2) P (3) (4) K (5) 3 <6) .12 (7) 94 (8) 971 

(9) (10) 1 (11) 1 (12) GP (13) 768 (14) LP (15) 

(16) (17) (18) (19) (20) (21) (22) 

6. IN ORDER TO OBTAIN A RANDOM SAMPLE 10 

1. SEVERAL HUNDRED PEOPLE MUST USUALLY BE SELECTED. 20 
** 2. EACH PERSON IN THE GROUP FROM WHICH THE SAMPLE IS TO BE DRAWN 30 

MUST HAVE AN EQUAL CHANCE OF BEING SELECTED. 40 

3. EVERY SUBCLASS IN THE POPLUATION MUST BE EQUALLY REPRESENTED. 50 

4. THE EXPERIMENTER SHOULD ASK FOR VOLUNTEERS. 60 
(1) 82 (2) ST (3) (4) K (5) 1 (61 .57 (7) 67 (8) 971 
(9) (10) 1 (11» I (12) GP (13) 768 (141 IP (15) 

(16) (17) (18) (19) (20) (21) (22) 
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FIGURE II 



MARCH 29, 1973 DELTA COLLEGE PAGE 



1. AMBIVALENCE ILLUSTRATES 

1. FRUSTRATION TOLERANCE* 

2. APPROACH-AVOIDANCE CONFLICT. 

3. APPROACH-APPROACH CONFLICT. 
4« AVOIDANCE-AVOIDANCE CONFLICT. 
5. 

2. THE DESIRE FOR AN OBJECT FOR ITS OWN SAKE IS KNOWN AS.. 

1. EXTRINSIC MOTIVATION 

2. POSITIVE TRANSFER 

3. STRESS INCENTIVE 

4. INTRINSIC MOTIVATION.. 

3. THE AMOUNT OF STRESS WHICH A PERSON CAN BEAR WITHOUT SHOWING BEHAVIORAL 
DISORGANIZATION IS HIS 

1. LEVEL OF ASPIRATION 

2. FRUSTRATION TOLERANCE 

3. AVOIDANCE LEVEL 

4. PHYSIOLOGICAL LIMIT 

5. POINT OF INTERESCTING GRADIENTS 

4. THE SYMPATHETIC AND PARASYMPATHETIC SYSTEMS ARE DIVISIONS OF THE 

1. PRIMARTY NERVOUS SYSTEM. 

2. AUTONOMIC NERVOUS SYSTEM. 

3. EPINEPHRINE-NOREPINEPHRINE SYSTEM. 

4. GENERAL NERVOUS SYSTEM. 

5. THE TWO HEMISPHERES OF THE CEREBRUM ARE CONNECTED BY 

1. THE BRAIN STEM. 

2. THE RETICULAR FORMATION. 

3. THE CORPUS CALLOSUM. 

4. A LARGE FISSURE. 

6. IN ORDER TO OBTAIN A RANDOM SAMPLE 

1. SEVERAL HUNDRED PEOPLE MUST USUALLY BE SELECTED. 

2. EACH PERSON IN THE GROUP FROM WHICH THE SAMPLE IS TO BE DRAWN 
MUST HAVE AN EQUAL CHANCE OF BEING SELECTED. 

3. EVERY SUBCLASS IN THE POPLUATIDN MUST BE EQUALLY REPRESENTED. 

4. THE EXPERIMENTER SHOULD ASK FOR VOLUNTEERS. 

7. A BELIEF THAT WE ACCEPT EVEN THOUGH IT CANNOT BE DEMONSTRATED SO 
CONVINCINGLY AS THE RELATIONSHIPS WE CALL FACTS IS CALLED 

1. AN IMAGE 

2. A PREMISE 

3. A SYMBOL 

4. A CONCEPT 
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File Maintenance 

Any record of a record group may be updated using card input. A complete record 
group,. or a question or answer record only can be deleted by the same program. 
Card Input Formats 

1.. Updating a Header Record ■ / 

cc 

1-5 Question Number 

6 Character 'u* 

7-10 Blank ' ' 

11-78 Various fielda to be updated^ 
79-80 Blank 

2. Updating the Answer Byte on the Header Record 
cc 

1-5 Question Number 

6- 10 Digits 1-5', or blank (if blank the answer code will contain no answer) 
11-78 Various fields to be updated^ 

79-80 Blank 

1 

3 . Deleting a Record Group 
cc 

1-5 Question Number 
6 Character 'd * 

7- 80 Blank 

4. Updating a Question or Answer Record 

Change the original question/answer card to the desired data - 

5. Adding a Question or Answer Record 

Use the question/answer card format with a card sequence number between any 
two card sequence numbers where the record is to be added. 

6. Deleting a Question or Answer Record 

Use all the question/answer cards except the ones to be deleted for any 
question number record group. , ' 

^If a field on the input card is blank, no change will occur on the disk file 
header record for that particular field. 



Evaluation 

Tests were initiall y printed in the Suinmer of 1968 by a very much simplified 
version of the test library concept. The system described in the paper was implemented 
about September 1971. The student acceptance of the test in terms of legibility of 
print-out was initially a problem. However, we have found that by using a new printer 
ribbon we can get copy directly from the printer which can be photographically reproduced 
with no problems in legibility. 

These tests were used exclusively in my General Psychology classes this last Fall 
semester and I received absolutely no student complaints on the legibility print, or 
the format of the questions. 

The system does not, however, eliminate in advance bad questions which causes 
legitimate student complaints. However, the system is designed so that the questions 
can be modified and these complaints over time should reduce to nearly zero, except 
for new questions which will add to the library. 

The difficulty with the system is primarily one of putting all of the codes into 
the compnter. This difficulty lies not with computer but with the work required of 
the instructor to do the coding. 

The acceptance by my colleagues of the Computerized Test Library concept and 
operating programs has not been good. The reasons for this, I think, are the fallowing: 

1. involves a tremendous amount of, work to do the coding and, 

2. a general fear and reluctance to use of computerized procedures. 

The real drawback in terms of adoption at other colleges and universities is that 
the test library is very expensive in computer time. This does not cause us any real 
problem since we own the computer outright. However, if one had to pay the standard 
rates for computer time the cost of the system might well exceed the practical benefits. 
As can be seen from Figure 1, we have not as yet been able to completely code the questions 
in the library. 

Currently a program is being developed to rearrange the position of the correct 
answer. This will insure that the correct answer will appear equally frequently in 
all choices. 
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In general it is my feeling that over a period of time the test library will 
rtsult in greatly improved questions and tests because of the data and ease of mani- 
pulating and selecting questions on the basis of that data. 
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